phase1: remove kmods in target packages if archive is enabled #37
Conversation
dangowrt
force-pushed
the
fix-kmod-archive
branch
from
b21b488
to
e558e9f
Compare
Do you've more details about the issue, ideally reproducer? I just tried following and don't see any issue. root@OpenWrt:/# ubus call system board
{
"kernel": "6.6.32",
"hostname": "OpenWrt",
"system": "ARMv8 Processor rev 4",
"model": "Linksys E8450",
"board_name": "linksys,e8450",
"rootfs_type": "initramfs",
"release": {
"distribution": "OpenWrt",
"version": "SNAPSHOT",
"revision": "r26463-a1a9572f43",
"target": "mediatek/mt7622",
"description": "OpenWrt SNAPSHOT r26463-a1a9572f43"
}
}root@OpenWrt:/# opkg update
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_core
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/base/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_base
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/base/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_kmods
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_luci
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/luci/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_packages
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_routing
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/routing/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_telephony
Downloading https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/telephony/Packages.sig
Signature check passed.root@OpenWrt:/# opkg install kmod-usb-serial
Installing kmod-usb-serial (6.6.32-r1) to root...
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/kmod-usb-serial_6.6.32-r1_aarch64_cortex-a53.ipk
Installing kmod-nls-base (6.6.32-r1) to root...
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/kmod-nls-base_6.6.32-r1_aarch64_cortex-a53.ipk
Installing kmod-usb-core (6.6.32-r1) to root...
Downloading https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/kmods/6.6.32-1-5ca3cc895cfde6fa5bcd4cac1657aceb/kmod-usb-core_6.6.32-r1_aarch64_cortex-a53.ipk
Configuring kmod-nls-base.
Configuring kmod-usb-core.
Configuring kmod-usb-serial.root@OpenWrt:/# dmesg | grep usbserial
[ 144.192904] usbcore: registered new interface driver usbserial_generic
[ 144.192952] usbserial: USB Serial support registered for generic |
|
@ynezz Reproducing is easy. Install a snapshot today and try installing a kmod package or anything depending on a kmod package in a few weeks, after a kernel version bump. Despite the kmod archive feed being present, opkg well prefer the no longer matching newer kmods instead of choosing the ones from the kmod archive. |
|
I can confirm this wrong opkg behavior and purging the kmods from the core repo will fix it. Basically, our opkg fork is broken when it has to pick a dependency with version constraints from multiple providers. Likely one of our opkg memory usage reduction patches is breaking this. |
|
(this problem with kmods and dependency was one of the main reason we wants to move to apk) |
|
I think we tried to tackle this issue back in 2020 and even made it working. I wonder what broke it again... |
|
I've just deployed this change "manually" to https://buildbot.staging.openwrt.org |
|
Long live the OPKG |
|
@Ansuel Thank you! You are too fast 😜 |
|
@dangowrt ehehehe problem is that i think this needs to be manually picked for staging buildbot :( |
|
Thanks, fix deployed. |
|
Lovely find command that loves order of args... @ynezz can you redeploy one last time :( ? |
Done |
ynezz
changed the title
|
Just added apk compat and printing of deleted files, deployed to staging. |
|
@ynezz in case you miss IRC i think an intermediate commit was deployed to staging |
Yep, should be fixed. I've as well (hopefully) fixed the wrong find expression. |
|
@ynezz I think i understood why this fail https://buildbot.staging.openwrt.org/images/#/builders/1/builds/8/steps/59/logs/stdio the file list is generated by the sha256sums file that is generated before the kmodclean is called (when the kmods are still in the target directory) hence rsync complain that the file can't be found. Solution is to move the step before the checksums call so that file gets correctly deleted and are not included in the final file. Can you check if this theory is correct and redeploy? If it does look ugly we may have to add an additional step to remove the missing entry in the sha256sums (if we want to keep everything in order) |
Ansuel
force-pushed
the
fix-kmod-archive
branch
2 times, most recently
from
fbd516a
to
a4d912e
Compare
|
Well it seems at the end we really need to ""fix"" the sha256sums... Alternative solution is to use a tmp directory to move the kmods. Honestly they are both bad solution but one should be faster than moving lots of files around one extra time... |
|
@ynezz any idea how to handle the lint problem with this? |
Ansuel
force-pushed
the
fix-kmod-archive
branch
from
d0ad509
to
25d364a
Compare
|
Just to assert the situation. We really need to fix this:
TODO:
|
Ansuel
force-pushed
the
fix-kmod-archive
branch
12 times, most recently
from
9b6a9f5
to
cf7a98b
Compare
Implement custom step ShellCommandAndSetProperty, as an extension of ShellCommand with the addition of setting a bool property that is set True or False if the shell command succeeded or not. Signed-off-by: Christian Marangi <[email protected]>
Ansuel
force-pushed
the
fix-kmod-archive
branch
from
cf7a98b
to
a439116
Compare
Ansuel
force-pushed
the
fix-kmod-archive
branch
2 times, most recently
from
a5a26ae
to
c50f1a4
Compare
OPKG gets confused if kmod packages are present in both, target packages as well as kernel version specific folder. Remove them from target packages to make opkg pick the kmods from kmod archive folder only. This also affects APK that picks packages from the first entry in the repository file and doesn't support checking the same package from multiple entry. Signed-off-by: Daniel Golle <[email protected]> [ add --remove-source-files and improve implementation ] Signed-off-by: Christian Marangi <[email protected]>
Ansuel
force-pushed
the
fix-kmod-archive
branch
2 times, most recently
from
4f6aa55
to
9fec5ce
Compare
Add additional logic to merge the sha256sums with the remote kmods entry (in remote sha256sums) already present. This is to produce a more consistent sha256sums that actually reflect what is present in the remote server by including also the sha of the kmods for each kernel version supported. Signed-off-by: Christian Marangi <[email protected]>
There is currently a BUG in the --exclude pattern for the targetupload step. The /kmods/ directory is actually never excluded. This was never notice as the rsynclist never actually had kmods entry as make checksum step was done before moving the kmods to the dedicated directory. This is caused by the fact that with --files-from, not only the /kmods/ directory needs to be excluded but also every content in it, hence the additional --exclude "/kmods/**" is required to correcly skip the entry present in rsynclist. Signed-off-by: Christian Marangi <[email protected]>
Handle case where a sha256sums might contain previous kmods/. In such case packages.adb for previous kmods/ needs to be skipped as not included in the sign tar and already signed by previous runs. Skip is limited to kmods/ entry as those are the only entry expected to be present in the sha256sums but not included in sign tar. Signed-off-by: Christian Marangi <[email protected]>
Ansuel
force-pushed
the
fix-kmod-archive
branch
from
9fec5ce
to
174e7a1
Compare
|
Thanks a lot, merged in 31eb26b |
OPKG gets confused if kmod packages are present in both, target packages as well as kernel version specific folder.
Remove them from target packages to make opkg pick the kmods from kmod archive folder only.
I didn't test this and I'm not very familiar with buildbot. Just noticing that currently, older snapshots fail to install packages because newer-version kmods from target packages are selected instead of the ones matching the kernel version, so this should be fixed somehow...